Library Integration for IMU / Example Application

From RidgeRun Developer Wiki




RidgeRun Video Stabilization Library
Table of Contents [Sticky]






Introduction

The following sections will explain the expected behavior and operation of the library's included example application. This application exercises mainly the integration, interpolation, and stabilization algorithms, while the sensor data and video footage are provided externally.

Concept: Video Stabilization with Gyroscopic Data

The application consists on two main parts: a preprocessing stage that emulates the inputs available in a real live feed, and the main loop that captures frames and gyroscopic samples as seen in the diagram below.

Block Diagram for Example Application
Block Diagram for Example Application

Next, a summary of the instances and stages is provided.

Instances

This application uses the following library components:

  • An Integrator with Integrator Settings.
  • An Interpolator with Interpolator Settings.
  • A Stabilizer with Stabilizer parameters.
  • A Fish Eye Undistort Instance with Null Parameters.

Preprocessing and Preparation

The input video will be downloaded from a remote link when building the project. At the same time, the gyro data is already included in the artefacts subdirectory for this project as a CSV file. The code will check that it is possible to:

  • Open the input video.
  • Open the gyro data file.
  • Create an "output.mp4" file for the final result.

For this example, the video properties, such as codec and framerate, are extracted from the input file, although the output resolution is fixed to 640x480.

If the previous steps are successful, the program advances into the main loop, which treats each frame.

Main Operation Loop

This loop emulates how a live video would be processed, taking into account that data is available only from the current and past frames.

Pre-Integration

For each frame, there is an unknown number of gyro samples that match. Therefore, the first step is accumulating samples for the current time interval. If the remaining samples were insufficient to fill the interval, it will be considered the video's last frame.

Integration

The integrator transforms the samples into their corresponding timestamp-quaternion pair representation in an accumulation buffer.

Post-Integration

After integration, the timestamps are transposed due to the delay between the frame's capture and measurement.

Interpolation

The interpolator will check if the accumulated buffer has enough samples before interpolating. If not, the example will return to the integrator and extend the buffer. once the interpolation is done, the accumulated buffer is cleaned for the next integration.

Post-Interpolation

An interpolation buffer will take the first sample of the interpolated output, and the previous steps will repeat until there are three samples.

Stabilization

The 3-sized interpolation buffer is used to get a stabilized buffer of the same size. The middle values of the stabilized buffer and the interpolation buffer are used to estimate the correction rotation.

Image Transformation

To correct the image, the input and output frames must be allocated using the image allocator included in the library and the appropriate backend (this example employs OpenCV); and then an undistort instance transforms the image using the rotation obtained from the stabilization step. Finally, the frame is written into the output video.

Additional Details

The application is constructed by default by the building system. Make sure of having -Denable-examples=enabled in the construction.

First, download the video and the raw gyro data using: 

wget "https://drive.usercontent.google.com/download?id=1uW2sg3E2W2UOF9rjDaHG8eJus7m531_4&export=download&authuser=0&confirm=f" -O test-video-reencoded.mp4

wget "https://drive.usercontent.google.com/download?id=1LXjqut2c8YIiJg66vH_UyYSdP664OErw&export=download&authuser=0&confirm=f" -O raw-gyro-data.csv

To execute, simply: 

BUILDDIR=builddir
./${BUILDDIR}/examples/concept/rvs-complete-concept -f test-video-reencoded.mp4 -g raw-gyro-data.csv -b opencv

For more options: 

Usage:                                                     
    rvs-complete-concept [-n NUMBER_OF_FRAMES] [-f INPUT_VIDEO] [-g GYRO_DATA]
             [-o OUTPUT_VIDEO] [-b BACKEND] [-w WIDTH] [-h HEIGHT]
             [-s FIELD_OF_VIEW_SCALE]      

Options:                                                   
    --help: prints this message                            
    -g: raw gyro data in CSV                               
    -f: input video file to stabilise                      
    -o: output video file stabilised. Def: output.mp4      
    -b: backend. Options: opencv (def), opencl, cuda       
    -n: number of frames to stabilise                      
    -w: output width (def: 1280)                            
    -h: output height (def: 720)                           
    -s: field of view scale (def: 2.4)

With this, it is possible to change the location of the files, select a backend and modify the output resolution for performance testing. The camera matrix is set by default and it may not lead to the most optimal results. Moreover, higher the resolution, you may need to lower the field of view scale (-s) to have zoom over the image.

If you are looking to compile your application and use our library, you can compile this example using: 

RAW_GYRO_FILE=FULL_PATH_TO_RAW_GYRO
VIDEO_FILE=FULL_PATH_TO_VIDEO
g++ -o concept complete-code-example.cpp $(pkg-config rvs --cflags --libs) -DRAW_GYRO_INPUT_FILE=${RAW_GYRO_FILE} -DTEST_VIDEO=${VIDEO_FILE}

Note: The library has to be installed previously.




Retrieved from "https://developer.ridgerun.com/wiki/index.php?title=RidgeRun_Video_Stabilization_Library/Library_Integration_for_IMU/Example_Application&oldid=61138"